Introduction to Machine Learning

Unit 04: Hold-Out and K-Fold Cross-Validation

1. Introduction

A model that performs perfectly on its training data tells us nothing useful. This unit teaches how to reliably estimate a model's true generalization performance, and how to use that estimate to pick the best hyperparameters. We will cover the distinction between model parameters and hyperparameters, the basic hold-out train/test split, the 3-way train/validation/test split for model selection, and the industry-standard K-Fold Cross-Validation.

Learning Objectives

2. Theory

2.1 Parameters vs. Hyperparameters

Parameters
Hyperparameters

Learned automatically from training data during fitting.

AlgorithmParameters
Linear / Logistic RegressionCoefficient vector β and intercept β₀
Neural NetworkWeights W and biases b of every connection
Decision TreeActual split conditions and thresholds at each node
KNN(None — KNN stores all training points directly)

Set by the user before training starts. They control the learning process itself.

AlgorithmHyperparameters
KNNk (neighbors), weights (uniform/distance), metric, p (Minkowski)
Neural NetworkLearning rate α, #layers, #neurons/layer, batch size
Decision Treemax_depth, min_samples_leaf, splitting criterion (gini/entropy)
Ridge/Lasso RegressionRegularization strength α

2.2 Hold-Out Method — Basic 2-Way Split

Full labeled dataset split into training and test sets Eighty percent of the labeled dataset is used for training and twenty percent is held out untouched until final evaluation. FULL LABELED DATASET 80% FOR TRAINING 20% HELD OUT TRAINING SET Used for: Fitting the model Learning parameters (β, θ, splits, etc.) TEST SET Locked in a box Evaluated ONCE at the END RULE Test set stays TRULY untouched until the FINAL evaluation.

The test-set accuracy is the held out. The test set error estimates generalization error (out-of-sample error) — how well the model will perform on truly unseen data. Low training error + high test error = overfitting.

The Hold-Out Flaw for Hyperparameter Tuning

If we reuse the test set repeatedly for model selection / hyperparameter tuning), it effectively becomes part of the training data and the model overfits to the test set. Scores become optimistic (inflated, misleading, not reproducible on truly unseen data.

2.3 The Three-Way Split — Train / Validation / Test

The fix: split three separate chunks, not two:

Full Labeled Dataset Split A diagram showing training, validation, and held-out test datasets with the model selection and final evaluation workflow. FULL LABELED DATASET ~64% TRAIN ~16% VALIDATION 20% TEST (HELD OUT!) Train Validation Test • Fit model • Learn params • Rank hyperparams by CV score Evaluated exactly ONCE after best model is selected! RETRAIN BEST CONFIG ON TRAIN + VAL then evaluate Model fitting Hyperparameter selection Final unbiased evaluation
  1. Training set (~64%): Fit different models with many hyperparameter values.
  2. Validation set (~16%): Evaluate each trained model; pick whichever hyperparameters perform best.
  3. Retrain the winning configuration on TRAIN + VAL combined.
  4. Test set (20%): Evaluate the final retrained model exactly once — that number is your reported generalization.

2.4 K-Fold Cross-Validation

The hold-out validation estimates are sensitive to exactly which rows landed in the validation split. K-fold fixes this by repeating the process k times on different partitions and averaging:

K-fold cross-validation workflow Training and validation portion of the data, showing iterations where each fold is used once as a test set and the remaining folds are used for training. Training + Validation Portion K-fold cross-validation partitions the available data into reusable training and test folds. 80 rows shown as 80% Fold structure The 80% training + validation portion is divided into k folds. Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 6 ⋯ Fold k Cross-validation iterations TEST fold TRAIN folds Fold allocation Validation score Iteration 1 TESTTRAINTRAINTRAINTRAINTRAIN⋯TRAIN score₁ Iteration 2 TRAINTESTTRAINTRAINTRAINTRAIN⋯TRAIN score₂ Iteration 3 TRAINTRAINTESTTRAINTRAINTRAIN⋯TRAIN score₃ ⋮ additional fold rotations ⋮ Iteration k TRAINTRAINTRAINTRAINTRAINTRAIN⋯TEST scoreₖ Final CV score Average the validation score from every fold rotation. average(score₁, score₂, score₃, …, scoreₖ)

K-Fold CV Steps

  1. Randomly partition the training + validation rows into k disjoint, equal-sized folds without replacement.
  2. For i = 1 … k: Train on folds 1..k except i; evaluate on fold i → get scoreᵢ.
  3. Report the mean of the k scores (and optionally their standard deviation) as the CV performance estimate.
  4. After hyperparameters are chosen via CV, retrain the final model on the entire train+val set.
  5. Do the final evaluation on the still-locked-away test set.

Standard value: k = 10 (10-fold CV) is the default in nearly every ML paper. Stratified k-fold (for classification) ensures each fold has roughly the same class distribution as the whole dataset.

2.5 Leave-One-Out Cross-Validation (LOOCV)

Special case: set k = n (the number of training examples). Each fold is a single row:

2.6 Accuracy Metric Review — The Confusion Matrix

Every evaluation in this unit uses the standard classification accuracy:

Predicted Label
Positive (+)Negative (−)
True
Label		+True Positive (TP)False Negative (FN)
−False Positive (FP)True Negative (TN)
\( \text{Accuracy} = \dfrac{TP + TN}{TP + TN + FP + FN} \)

2.7 sklearn cross_val_score and n_jobs

Scikit-learn makes k-fold CV one line:

from sklearn.model_selection import cross_val_score # KNN with K=19, stratified 10-fold CV, accuracy scores = cross_val_score(model, X_trainval, y_trainval, cv=10, scoring='accuracy', n_jobs=-1) print("CV scores per fold:", scores) print("Mean CV accuracy: %.3f (SD %.3f)" % (scores.mean(), scores.std()))

n_jobs parallelizes across CPU cores: n_jobs = 1 → sequential; n_jobs = 2 → two folds at once on 2 CPUs; n_jobs = −1 → all available CPUs.

3. Interactive Examples

Example 1: Parameter or Hyperparameter?

Classify each item. Reveal per item by expanding.

(A) In a KNN classifier: "the maximum depth used when the split threshold of a decision tree.

Reveal
Both of the first (KNN's k), the maximum tree splits split conditions are hyperparameters. The user sets them before training. The tree's learned splits are parameters; the max depth restriction is a hyperparameter.

(B) coefficient β₁ of a linear regression: the slope on the income.learned" logistic regression weight."]parameters" coefficient.

Reveal
The coefficient β₁ and logistic regression> they are not set by the user; are parameters. They are learned directly from data during training. The regularization α that penalizes them is a hyperparameter.

(C) learning rate of gradient descent during neural network training."]

Reveal
Learning rate α is a hyperparameter. It controls the speed of convergence but is not learned from data — you try several and pick the best via CV.

Example 2: Train / Val / Test Split Decisions

Data Leakage Detective

Five mini-scenarios. For each, answer: is this allowed ML practice, or does it leak data / invalidate the test score?

  1. "After training my dataset (a): I use the test set accuracy to decide between k=3 and k=5 neighbors."
  2. "(b): I use 10-fold CV on TRAIN+VAL to pick k, then report the k winner kNN on the full train+val, evaluate score the final score on held-out test."
  3. "(c): StandardScaler on the WHOLE dataset first, then split into train/test."
  4. "(d): Split first, StandardScaler().fit(X_train) only, then transform X_train and X_test."
  5. "(e): I cross_val_score, StandardScaler inside a Pipeline wrapping both scaling + KNN, then fit the pipeline on each fold separately per CV training splits."
  1. Leakage! Picking k on the test set overfits to test.
  2. ✅ Correct. Classic 3-way split protocol.
  3. Leakage! Scaling fit is done on whole dataset → test-set mean/std pollutes the training preprocessing.
  4. ✅ Correct. fit on train only; transform both. This is how it's done.
  5. ✅ Best practice. Pipeline + CV together prevent leakage during the cross-val folds themselves.

Example 3: Manual 3-Fold CV by Hand

Click to see tiny dataset. Compute 3-fold CV accuracy by hand for a trivial 1-NN.

Six labeled 1-D points with classes: {X=[1,2,3,4,5,6] and y=[A,B,A,B,A,B].

Fold 1 = rows {1,2}, Fold 2 = {3,4}, Fold 3 = {5,6}. Use KNN with k=1. Compute per-fold accuracy, then mean CV accuracy.

(i) Iteration 1: train on [X=[3,4,5,6] / y=[A,B,A,B]; test on {1: A, 2: B}. 1-NN classifies test[1] using nearest [2(B), 1→B (predict A is mispredict! Wait — recheck: X=1 nearest is 2 (B) → predict B but true y=A. Error. X=2 nearest is 1(A) → predict A, true=B → error. Accuracy fold 1 = 0 / 2 = 0.0.

(ii) Iteration 2: train on [1,2,5,6] y=[A,B,A,B]; test {3:A, 4:B}. X=3 nearest is 2(B)→B≠A→wrong; X=4 nearest 5(A)→A≠B→wrong. Accuracc = 0.0.

(iii) Iteration 3: train [1,2,3,4] y=[A,B,A,B]; test {5:A, 6:B}. X=5 nearest 4(B)→B≠A; 6 nearest 5(A train points to nearest 4(B)=predict B≠A. Accuraccy=0.0.

Mean CV accuracy = (0 + 0 + 0)/3 = 0.0. (The dataset alternates A/B/A/B in 1D so k=1 literally always guesses wrong.)

4. Numerical Solutions

Problem 1: Split Arithmetic from the Confusion Matrix

A binary classifier on 1,000 test samples produces: TP = 120, FN = 60, FP = 40, TN = 780.

  1. Build the 2×2 confusion matrix and verify the totals add up.
  2. Compute classification accuracy.
  3. Compute True Positive Rate (Recall/Sensitivity) and False Positive Rate.
📘 Step-by-step

(a) Confusion matrix:

PredictedTotal
+−
True+120 (TP)60 (FN)180
−40 (FP)780 (TN)820
Total1608401000

(b) Accuracy = (120 + 780)/1000 = 0.900 (90%).

(c)

\( TPR = \frac{TP}{TP+FN} = \frac{120}{180} = 0.\overline{6} \approx 66.7\% \) \( FPR = \frac{FP}{FP+TN} = \frac{40}{820} \approx 4.88\%

Problem 2: K-Fold on Small Dataset

Small n = 150 labeled training rows. Perform a stratified k = 5 stratified CV.

  1. How many rows are test in each fold?
  2. In each iteration, how many rows are used for training?
  3. How many distinct models are fitted in total?
  4. Suppose the fold accuracies are {0.87, 0.90, 0.83, 0.87, 0.93}. Report the 5-fold CV mean accuracy and its standard deviation (sample).
📘 Step-by-step

(a) 150 / 5 = 30 rows per fold.

(b) 150 − 30 = 120 training rows per iteration.

(c) 5 folds → 5 separate model fits → 5 models. (Then +1 final refit on all 150 rows once hyperparameters are chosen, for a total of 6 fits.)

(d)

\( \bar{x} = \frac{0.87 + 0.90 + 0.83 + 0.87 + 0.93}{5} = \frac{4.40}{5} = \mathbf{0.88} \) \( \text{Sample SD} = \sqrt{\frac{(−0.01)^2 + (0.02)^2 + (−0.05)^2 + (−0.01)^2 + (0.05)^2}{4}} = \sqrt{0.0014} \approx \mathbf{0.0374}

Report: CV accuracy = 88.0% (± 3.7%).

Problem 3: 10-Fold vs. LOOCV on n = 81 samples

  1. How many models total model fits does 10-fold require?
  2. How many for LOOCV?
  3. Give one statistical advantage of 10-fold: 1000-labeled dataset. (b) n=15 dataset?
📘 Step-by-step

(a) 10 → 10 fits (plus 1 final refit = 11).

(b) LOOCV = n = 81 rows → 81 fits (plus 1 refit → 82 total).

(c) (i) n=1000: 10-fold clearly better speed wins — 10 models instead of 1000 models, plus 10-fold scores are a nice (each fold ≈ 900 train, which is plenty); LOOCV would be overkill. (ii) n=15: 10-fold leaves only 1–2 test per fold — scores unreliable. LOOCV trains on 14 rows, tests 1, no randomness → better estimate for tiny datasets prefer LOOCV!)

5. Try It Yourself

Problem 1 — 3-Way Split Sizes

Dataset of 2,500 rows. Use 64/16/20 train/val/test split.

  1. How many samples land in each split?
  2. Which split is used to pick k?
  3. After k is picked, which split(s) do you retrain the final model on?
  4. Which split do you evaluate the final retrained model on, and how many times?

(a) 2500 × 0.64 = 1,600 train; × 0.16 = 400 val; × 0.20 = 500 test.

(b) The validation set — or via k-fold on the combined train+val (2,000 rows).

(c) Train + Validation combined (2,000 rows).

(d) Evaluate exactly once on the 500-row test set. One single number — that is the reported generalization accuracy.

Problem 2 — K-Fold CV with sklearn Workflow

You compare four KNN classifiers on a binary classification task: k ∈ {3, 7, 15, 31}. 5-fold CV gives fold accuracies below:

kFold1Fold2Fold3Fold4Fold5
30.850.820.880.800.85
70.890.860.900.870.88
150.880.890.860.910.91
310.820.830.810.840.85
  1. Calculate mean CV accuracy per k and pick the best k.
  2. Compute sample SD of CV scores for both k = 7 and k = 15. Which is more stable?
  3. After picking the best k, describe in 1–2 sentences what you do next.

(a) Means:

  • k=3: 4.20/5 = 0.840
  • k=7: 4.40/5 = 0.880
  • k=15: 4.45/5 = 0.890 ← best mean
  • k=31: 4.15/5 = 0.830

(b) SD(k=7): values =0.880 → devs [+0.01,−0.02,+0.02,−0.01, 0.00] → var =0.00025 → SD =0.0158. SD(k=15): mean=0.890 → devs [−0.01, 0, −0.03, +0.02, +0.02] → var=0.00045 → SD ≈ 0.0212. k=7 slightly more stable; both good. Winner k=15 wins by mean accuracy.

(c) Retrain a single KNN(k=15) classifier on the full TRAIN+VAL combined dataset, then evaluate exactly once on the held-out test set. Report that single value as your generalization accuracy.

Problem 3 — LOOCV on Tiny n=4

4 points: X=[1,2,4,5]; y=[A,A,B,B]. 1-NN classifier. Compute LOOCV accuracy.

Iter 1: test 1 (A). Train on {2(A),4(B),5(B)}. Nearest of 1 is 2(A) → predict A correct ✔

Iter 2: test 2 (A). Train {1(A),4(B),5(B)}. Nearest is 1(A) → predict A correct ✔

Iter 3: test 4 (B). Train {1(A),2(A),5(B)}. Nearest is 5(B) → predict B correct ✔

Iter 4: test 5 (B). Train {1(A),2(A),4(B)}. Nearest is 4(B) → predict B correct ✔

LOOCV accuracy = 4 / 4 = 1.00 (100%).

6. Interactive Quiz

Answer all 5 MCQs. Click on an option to get instant feedback.

Your score: 0 / 5

7. Key Takeaways

  1. Parameters are learned; hyperparameters are set. Don't confuse them — parameters come from data; the other requires tuning via CV.
  2. Two splits: Train (64/16/20 train/val/test. Never pick hyperparameters on val; never report final performance on test.
  3. Never tune hyperparameters on the test set. It leaks data; gives optimistic scores. Use validation (or CV on trainval).
  4. K-fold CV gives a less-biased, lower variance estimate. Repeat hold-out across k folds, average the k fold scores. k=10 is the standard default.
  5. Stratified k-fold. Use for classification to preserve each fold's class distribution.
  6. LOOCV = k = n. Use only for very small datasets; expensive but deterministic.
  7. Retrain winner on all trainval after picking via CV. Then evaluate once on test. That single number is your paper-ready result.

8. Common Pitfalls

  1. Test-set reuse = data leakage. Each time you peek at test accuracy and change the model, test information flows backward. Stop — use CV on the trainval set instead.
  2. Scaling before splitting. StandardScaler fit on whole dataset uses test-set mean/std. Fix: split → fit(train) → transform(train), transform(test).
  3. No Pipeline inside CV. If you scale the entire dataset once outside cross_val_score, each fold's training portion sees the validation fold's scaling statistics. Fix: use a Pipeline(StandardScaler → KNN) so each CV fold learns its own scaler from only that fold's training rows.
  4. Choosing k = 2 (binary ties). Use odd k to avoid 50/50 ties.
  5. LOOCV on n = 10 000. 10 000 fits would take days. 10-fold is essentially as accurate and 1000× cheaper.
  6. Reporting CV accuracy as the final number. CV chooses the hyperparameters. Final number is test accuracy after retrain on full trainval on that winner.

9. Resources